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"GSFC FSB Application of Perspective-Based Inspections": 

Overview 

PoC: Elaine Shell (Head, Flight Software Branch) 

Introduction 

The “GSFC FSB Application of Perspective-Based Inspections” Research Infusion Collaboration was 
performed by the Fraunhofer Center - Maryland under Grant NAG5 12556. The Project was a joint effort at 
the National Aeronautics and Space Administration (NASA) Goddard Space Flight Center with participation 
from the Flight Software Branch, including Branch Head Elaine Shell, Mike Tilley, and other personnel. 

The “GSFC FSB Application of Perspective-Based Inspections” Final Report describes the collaboration and 
documents the findings, including lessons learned. 

Problem statement 

Our primary objective was to produce Branch-baselined process standards for software inspections, which could 
be used across three new missions within the Flight Software Branch. The standard was developed using the 
Perspective-Based Inspection approach, then tested on a pilot Branch project. Based on our experiences on the 
pilot, we would decide whether the standard was suitable for roll-out to other Branch projects. 

The relatively short time scale for this collaboration meant that we would not be able to directly measure the 
reduction in rework effort (expected because improved defect detection guidelines will result in increased phase 
containment). Instead, within this timeframe we relied on a more qualitative measure: Whether or not the 
standard received high ratings from Branch personnel as to usability and overall satisfaction. 

The project used for piloting the tailored Perspective-Based inspection approach was the Core Flight Executive 
(cFE), a reusable multi-mission framework that will provide a flight software (FSW) operating environment and 
a set of services that host and support FSW applications. 

Application of the technology to the target project 

The perspective-based approach was applied to produce an inspection procedure tailored for the specific quality 
needs of the Branch. The technical information to do so was largely drawn through a series of interviews with 
Branch personnel, each of less than an hour. As a result, a set of the key perspectives was created (i.e. which 
technical viewpoints need to be represented during the inspection), along with a specific quality focus for each. 

Due to the relative complexity of the Perspective-Based inspection standard produced and the desire to get buy- 
in from Branch personnel, we did not enforce the use of the standard all at once. Instead, we advocated its use 
over a series of cFE requirements inspections. At each inspection, a member of the technology infusion team 
was present and charged with understanding which parts of the standard were being followed and which were 
dropped, and observing the effects of both aspects. Observing its use over a series of inspections gave us an 
opportunity to allow the Branch personnel to not feel overly restricted in using the standard but to follow the 
recommendations that made sense to them and then to understand the effects of following or not following the 
various guidelines. Over time, we have seen more and more of the guidelines adopted by the cFE inspection 
team and feel we are well-positioned for further usage by our three target Branch projects. 

There were no negative effects of the technology that in any way impacted our ability to achieve our cFE 
project deliverables. On the contrary, the inspections were useful for showing problems with our current 
approach on the cFE pilot project, which led to changes in the development project plan. 

Data Collection and Analysis 

Over the time period of this grant, we relied primarily on qualitative measures: How Branch personnel rated the 
standard, and whether it was adopted at the Branch level. We also collected quantitative data for monitoring 
whether the effects of the perspective-based technique were within acceptable limits. We intend to continue our 
collaboration with the technology infuser (Dr. Shull) past the end of this grant, to allow a more rigorous 
quantitative evaluation to be done. 
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Qualitative Measures. Did the development team (branch head and project representatives) give a positive 
rating for products delivered by the technology infusion team? This is by nature a subjective, qualitative 
measurement. The standard produced during this collaboration went out to Branch project representatives in 
two rounds of reviews, which yielded the following: 

• In the first round, the eight Branch personnel who reviewed the perspective-based standard had no changes 
to request. One said: “I really like the guide and wish I had it for past reviews!” 

• In the second round, the standard was disseminated more broadly; feedback identified important changes to 
be made. Analysis by Branch personnel of the feedback revealed that the primary issue was that the 
perspective of Attitude Control System (ACS) developers was not adequately represented. We feel 
however that this problem was a result of the specific personnel chosen for Dr. Shull to interview, in which 
ACS personnel were not represented, rather than a deficiency in the inspection approach itself. We feel that 
this deficiency is easily correctable. 

For what it’s worth, we also report that our independent CMMI lead assessor was extremely positive about the 
inspection standard and asked to take it as an example of good process documentation. 

Were the guidelines produced by the technology infusion team adopted as Branch standards? At this time, we 
have not included the perspective-based standard as part of the approved Branch standards since 1) Some 
rework was identified as needing to be done (described above), and 2) We decided that there was a need to 
better synchronize the roll-out of new Branch standards, including the inspection standard as well as others 
outside the scope of the technology infusion project. 

However, we in the Branch see no problems with the inspection standard itself that would prevent its roll-out 
and we expect it to happen in the near future. As Branch Head, I have been involved in reviewing the standard 
during several phases of its construction and am quite happy with the overall direction and the progress to date. 

Quantitative Measures. We also collected quantitative metrics during the four cFE requirements 
inspections in which the perspective-based standards were applied. We determined that the inspection rate 
during the meetings (i.e., how much of the requirements document could be inspected in a given time) met 
typical norms for NASA project environments (as specified by the “NASA Formal Inspections Training,” 
created by Dr. John Kelly). The above data shows that the use of the perspective-based standard developed 
on this project does not impose additional costs in terms of increased meeting time. 

We analyzed the Requests for Action (RFAs) generated by the requirements inspections to understand in which 
areas the defects were concentrated. Defects of “missing functionality” were by far the largest category; defects 
of “extraneous” (i.e. over-specified) functionality were also found. Together, these defects made up 75% of the 
total found. This result makes for an interesting comparison with an earlier analysis we did of change requests 
on previous Branch projects, in which missing/extraneous functionality formed a small minority of the total set. 
This result is consistent with the unique difficulties of the cFE project (namely, designing a system for 
maximum reusability rather than to support a specific mission), and may indicate corroboratory evidence that 
the inspection techniques were able to address the quality needs of the project. 

Summary and Lessons Learned 

The upfront tailoring process that was used in the perspective-based approach turned out to be crucial. The 
small effort spent on interviews and past defect history seemed to help improve effectiveness and relevance and 
help get buy-in from Branch personnel - by showing that the guidelines were relevant for Branch concerns. 

Although the technique has not yet become a Branch standard (for reasons unrelated to the technique itself), it 
will be the core of whatever standard we do eventually adopt. We feel that we need only an incremental fixing 
process to make the current perspective-based standard suitable for adoption by the Branch. 

We have already addressed the primary challenge we encountered, which was due to not fully considering all of 
the technical perspectives that really should have been included. We note that the perspective-based approach 
does seem to have been effective at achieving the buy-in of the Command and Data Handling (C&DH) 
personnel who were included in the upfront work. The lesson to be learned from this, we feel, is that it is 
extremely crucial to consider all interested stakeholders in the tailoring process, and to check whether 
assumptions about which technical roles have similar interests in the inspection process are in fact true. 



Research Infusion Final Report - "GSFC FSB Application of Perspective-Based 
Inspections" 

Table of Contents 

1.0 INTRODUCTION 5 

1 . 1 Problem Statement 5 

1.2 Target Project 5 

1.3 Collaboration Scope 6 

1 .4 Application of the technology to the target project 7 

2.0 METHODOLOGY 8 

3.0 DATA COLLECTION AND ANALYSIS 10 

4.0 SUMMARY AND LESSONS LEARNED 13 

5.0 ACRONYMS AND DEFINITIONS 15 


4 



Research Infusion Final Report - "GSFC FSB Application of Perspective-Based 
Inspections" 


1.0 Introduction 

The “GSFC FSB Application of Perspective-Based Inspections” Research Infusion 
Collaboration was performed by the Fraunhofer Center — Maryland under Grant NAG5 
12556. The Project was a joint effort at the National Aeronautics and Space Administration 
(NASA) Goddard Space Flight Center with participation from the Flight Software Branch, 
including Branch Head Elaine Shell, Mike Tilley, and other personnel. 

The “GSFC FSB Application of Perspective-Based Inspections” Final Report describes the 
collaboration and documents the findings, including lessons learned. 


1 . 1 Problem Statement 

The primary objective of the Project was to produce Branch-baselined process standards for 
inspection-related activities, which could be used across three new missions within the Flight 
Software Branch. A pilot application of these standards had the objectives of: 

> Showing that the Perspective-Based approach could incorporate relevant lessons 
learned from past Branch inspections and defect history; 

> Demonstrating the usefulness and feasibility of the Perspective-Based standards to 
Branch personnel and obtaining their buy-in; 

> Using the experiences from the pilot to refine the standards and increase their 
effectiveness. 

As discussed in our original proposal, the relatively short time scale for this collaboration 
meant that we would not be able to measure the reduction in rework effort (expected' because 
improved defect detection guidelines will result in increased phase containment - i.e. defects 
are less likely to slip from one phase to the next, and are much more likely to be caught 
before the testing phase, when they are the most expensive to correct). Instead, within this 
timeframe we relied on a more qualitative measure: Whether or not we received high ratings 
from Branch personnel as to the usability of the standards, and their satisfaction with the 
output produced. 

A secondary objective of our collaboration was to define and put in place the necessary 
metric collection mechanisms, which would allow us to undertake a more rigorous 
quantitative analysis of the inspection techniques over the lifetime of the pilot project. 


1.2 Target Project 1 

The project used for piloting the tailored Perspective-Based inspection approach was the 
Core Flight Executive (cFE). The cFE is a reusable framework that will provide a flight 
software (FSW) operating environment and a set of services that host and support FSW 


1 Most of the information in this section is taken from the Core Flight Executive requirements document, 
written by Dave McComas. 
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applications. By providing a multi-mission general-purpose software solution that supports 
variable hardware platforms, cFE will allow a broad range of future FSW applications to be 
built with less effort. 

Since cFE is a general-purpose software framework, not designed for a particular 
mission, cFE has all of the usual challenges faced by NASA software development 
efforts as well as some unique ones. First, the cFE must specify up-front requirements for 
FSW functions that have traditionally been left to the designers. Second, the cFE contains 
many interfaces that are intended to be standardized for multiple missions. Finally, 
certain functionality to be included in the cFE requirements contain mission-defined 
and/or optional features. 

Validating cFE through testing would require the development of a wide range of test 
applications and could only be done relatively late in the lifecycle. For this reason, the use of 
early-lifecycle inspections was felt to be especially crucial for this project with respect to 
exposing multi-mission assumptions and constraints to effective scrutiny. 


1.3 Collaboration Scope 

The scope of work described in our proposal consisted of developing inspection standards 
targeted to Branch-specific types of defects (gained from analysis of Branch project defect 
histories), and including Branch-relevant perspectives and questions to guide defect 
detection. The tailored inspection guidelines were to be applied on real Branch projects with 
support as needed from the technology infusion team. This still accurately describes the 
scope of work performed. 

It was originally proposed that the Perspective-Based inspection standard would be applied 
on three projects within the Branch: GPM, JWST, and SDO. Rather than apply the proposed 
standard to all three, we inserted a new step, in which the standard was instead applied on a 
single pilot project, cFE (described above). This decision was a good match for the Branch 
goals since, due to the “design for reuse” nature of cFE, inspections played an even more 
crucial than usual role in that development process. Also, since cFE is being designed to 
provide general-purpose functionality, key representatives from our target projects were 
involved in inspections of cFE to provide perspectives from different missions. In this way, 
they could get some exposure to and the chance to provide feedback on the proposed 
standards before applying them on their own projects. The Branch-baselined standards will 
still be applied on GPM, JWST, and SDO, although outside the time frame of this funding. 

Finally, we originally proposed using the analysis of Branch defect sources to indicate in 
which phases Perspective-Based inspections could provide the best potential for future 
improvement, although experience on previous Branch projects suggested that our efforts 
would likely be focused on requirements and code inspections. In the actual work, we 
focused exclusively on requirements inspections, as this was the highest-priority work 
currently being done on our cFE pilot project. 

As stated in the original proposal, due to the short timeframe of this collaboration, benefits 
due to the technology infusion are being measured via: 
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o Development team (branch head and project representatives) giving a positive rating 
for products delivered by the technology infusion team; 
o Adoption of guidelines produced by the technology infusion team at the project level, 
o Inclusion of the final, tested guidelines produced by the technology infusion team for 
use in the Branch standards. 

o The use by project personnel of data collection mechanisms suggested by the 

technology infusion team, so that the quantitative project metrics can be analyzed at a 
later date. 

However, over the lifecycle of the Branch projects (outside the timeframe of this funding), 
mechanisms have been put in place to measure benefits in (1) lower defect density of 
software artifacts throughout the lifecycle, and hence (2) fewer defects found during testing 
and (3) less time spent on reworking defects. 


1.4 Application of the technology to the target project 

Due to the relative complexity of the Perspective-Based inspection standard produced and the 
desire to get buy-in from Branch personnel, we did not enforce the use of the standard all at 
once. Instead, we advocated its use over a series of cFE requirements inspections. At each 
inspection, a member of the technology infusion team was present and charged with 
understanding which parts of the standard were being followed and which were dropped, and 
observing the effects of both aspects. Observing its use over a series of inspections gave us 
an opportunity to allow the Branch personnel to not feel overly restricted in using the 
standard but to follow the recommendations that made sense to them and then to understand 
the effects of following or not following the various guidelines. Over time, we have seen 
more and more of the guidelines adopted by the inspection team and feel we are well- 
positioned for further usage by our three target Branch projects: GPM, JWST, and SDO. 

The remainder of this section is organized around the specific points we were asked to 
address in this report. 

a. Did the project proceed according to the plan, including schedule? 

No schedule delays were introduced due to using the Perspective-Based inspections. 
However, due to the results of the inspection meetings it was decided to put more Branch 
personnel effort into a restructuring of the requirements document, in addition to making the 
more specific changes that were also suggested by the inspection team members. 

b. How much time was required to introduce and apply the technology? 

The majority of extra development team effort was spent on the upfront tailoring of the 
technology to the Flight Software Branch, as described in Section 2.0. The extra effort 
imposed for the introduction of the technique was limited to the inspection participants 
reviewing the documentation of the inspection standard. (Since Branch personnel already had 
experience with non-Perspective Based inspection processes, we did not feel that a dedicated 
training session was required.) Branch personnel began applying the Perspective-Based 
approach in the cFE inspections, with observations and recommendations made afterwards 
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by the technology infusion team. These inspections did not take appreciably longer than ones 
using a more traditional approach (see Section 3.0). 

c. What risks, including those you specified in Section 7, Management Plan, of your 
proposal, plus any others) needed to be addressed, what mitigation strategies were applied, 
and were they successful? 

The primary risk identified in the proposal was that too much effort might be required by the 
development team for non-development tasks. This threat was mitigated by 1) Allowing the 
infusion team to provide suggestions based on ongoing use of the Perspective-Based standard 
on real inspections of real development artifacts, rather than taking upfront time for training; 
2) Not enforcing 100% process conformance to the standard in the beginning, but allowing 
Branch personnel to adopt those recommendations over time. 

d. Evaluate the responsiveness of the technology vendor. Did the vendor provide the materials, 
training and user support in a timely manner? 

The technology infuser was appropriately responsive. The Branch was happy that Dr. Shull 
was able to adapt the work to the scheduling needs of the Branch and the pilot project. 

e. Were you successful in meeting your deliverables on schedule? If not, indicate why not. 

Regarding the planned deliverables of the technology infusion, the Branch and research 
infusion team jointly decided to change the deliverables and due dates to better suit the needs 
of the Branch and pilot project. (See Section 3.0 for detailed information on this.) 

Regarding the deliverables of the pilot development project, no project deliverables were 
missed due to any negative effects of the technology. Some of the inspections were useful for 
showing problems with our current approach on the cFE pilot project, which led to changes 
in the development project plan. 


2.0 Methodology 

Broadly speaking, the Perspective-Based inspection approach was first tailored to the FSB, 
then codified in a suitable format for a Branch standard, then introduced and piloted on cFE. 
Our collaboration proceeded through the following activities: 

1 . Reviewed Branch data collection guidelines and recommended changes based on data 
that would have to be collected throughout the project lifecycle to adequately evaluate 
the research technology. To do so, Dr. Shull reviewed our procedures for handling 
Discrepancy and Change Reports on Branch software development projects, to see if 
there were any metrics needed to evaluate the inspection techniques that were not 
already being collected. 

2. Analyzed Branch-specific defect sources, from project defect histories, in order to 
better focus the inspection techniques. Dr. Shull and his colleagues at FC-MD 
analyzed the change reports generated over the development lifetime of Swift BAT, a 
previous Branch project. This gave us a list of the types of problems typically seen on 
Branch projects, and hence an initial set of defect types on which the inspection 
procedure should focus. 
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3. Proposed a first set of Branch- specific perspectives, identifying key stakeholders who 
should be represented in any inspection in the Branch, that should be accounted for in 
the standard. This was done via initial conversations among Branch representatives 
Elaine Shell (Branch head) and Mike Tilley, other GSFC personnel (Mike Stark), and 
Forrest Shull, representing the technology infuser. These conversations resulted in 
identifying two initial perspectives: developer and tester. 

4. Refined the set of perspectives and targeted defect types through interviews with 
Branch personnel. This was done through five interviews, each of less than an hour, 
with personnel recommended by Elaine Shell. The result was to add a third 
perspective, representing experts in hardware or other product interfaces, and to add 
detail to the inspection guidelines for each perspective. 

5. Finalized the first draft of the inspection standard, through revisions suggested by 
Elaine Shell. 

6. Provided the standard to Branch personnel for comment. 

7. Held four cFE inspection meetings, with a member of the “infusion team” (either 
Forrest Shull or Mike Stark) present at each. In all cases Requests for Action (RFAs) 
were obtained and analyzed afterwards. 

8. Analyzed RFAs generated during the inspections to analyze defect sources, for future 
improvements to the focus of the inspection techniques. 
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3.0 Data Collection and Analysis 

This section is organized around the specific points we were asked to address in this report. 

a. What benefits were observed, including, but not limited to, those described in Section 8, 
Metrics, of your proposal? 

Qualitative measures. 

The metrics described in Section 8 of our proposal as suitable for the timeframe of this 
collaboration were introduced in Section 1.3 of this report. Here we describe our experiences 
in each of those areas. 

Development team (branch head and project representatives) giving a positive rating for 
products delivered by the technology infusion team. This is by nature a subjective, qualitative 
measurement. The standard produced during this collaboration went out to Branch project 
representatives in two rounds of reviews, which yielded the following: 

- In the first round, of the eight Branch personnel asked to review the Perspective- 
Based inspection standard, none had any changes to request. One said: “I really 
like the guide and wish I had it for past reviews!” 

In the second round, the standard was disseminated more broadly; feedback identified 
important changes to be made. Analysis by Branch personnel of the feedback revealed 
that the primary issue was that the perspective of Attitude Control System (ACS) 
developers was not adequately represented. We feel however that this problem was a 
result of the specific personnel chosen for Dr. Shull to interview, in which ACS personnel 
were not represented, rather than a deficiency in the inspection approach itself. We feel 
that this deficiency is easily correctable. 

For what it’s worth, we also report an additional qualitative response from personnel not employed 
directly by the Branch: Our independent CMMI lead assessor was extremely positive about the 
inspection standard and asked to take it as an example of good process documentation. 

Adoption of guidelines produced by the technology infusion team at the project level. 
Inclusion of the final, tested guidelines produced by the technology infusion team for use in 
the Branch standards. At this time, we have not included the perspective-based standard as 
part of the approved Branch standards since 1) Some rework was identified as needing to be 
done (described above), and 2) We decided that there was a need to better synchronize the 
roll-out of new Branch standards, including the inspection standard as well as others outside 
the scope of the technology infusion project. 

However, we in the Branch see no problems with the inspection standard itself that would 
prevent its roll-out and we expect it to happen in the near future. As Branch Head, I have 
been involved in reviewing the standard during several phases of its construction and am 
quite happy with the overall direction and the progress to date. 
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The use by project personnel of data collection mechanisms suggested by the technology 
infusion team, so that the quantitative project metrics can be analyzed at a later date. Dr. 
Shull analyzed the current Branch policies for Discrepancy and Change Reports in May 2004 
to ensure that defects and problems found in downstream development phases could be 
traced back to their starting point. A few minor modifications were recommended and the 
forms and database updated as a result. As a result we will be able to trace any future 
problems on the project to their origin and understand if they should have been caught by the 
Perspective-Based inspections or not. Having these data collection mechanisms in place 
allows us to develop a more quantitative and rigorous evaluation of the Perspective-Based 
inspection standard. 

Quantitative measures. We also collected quantitative metrics during the cFE requirements 
inspections in which some (although not all) of the perspective-based standards were applied. 
We determined that the inspection rate during the meetings (i.e., how much of the 
requirements document could be inspected in a given time) met typical norms for NASA 
project environments: Based on literally hundreds of inspections at JPL and GRC, the 
recommended rate for NASA requirements inspections is 10 to 30 pages in a 2-hour 
meeting. 2 The four inspection meetings for cFE covered: 

> 18 pages in 3 .5 hours 

> 14 pages in 3 hours 

> 12 pages in 2.5 hours 

> 12 pages in 2.5 hours 

The above data shows that the use of the standard developed on this project does not impose 
additional costs in terms of increased meeting time. The page rate for each inspection was 
about 5 pages per hour, at the low end (although not outside) of the recommended range of 5 
to 15 pages per hour cited above. This was likely a combination of getting used to the 
inspection procedures as well as the unusual nature of the cFE project (i.e., designing for 
reuse). Much of the time in earlier inspections was spent discussing the range of possible 
future missions that should be supported by cFE, which was a necessary prerequisite before 
the requirements themselves could actually be reviewed. Over time, based on earlier 
experiences, we made more of an effort to keep inspection meetings to around two hours 
duration, in line with the standard. 

The cFE requirements inspections also resulted in major technical defects being recorded in 
the following categories: 

> 18 of type “missing functionality”: Important functionality that should be provided by 
the cFE has not been described. 

> 4 of type “extraneous functionality”: Functionality was described that is likely not 
needed by future applications and/or is infeasible to provide. 

> 4 of type “data format”: The definition of the format of data handled by the cFE is 
either missing or incorrect with respect to knowledge about application needs. 

> 2 of type “incorrect functionality”: The functionality as described in the document is 
not likely to be appropriately reusable for future applications. 


2 Documented in the “NASA Formal Inspections Training,” developed by Dr. John Kelly. 
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> 1 of type “inconsistency”: Functionality or formats for similar items are not 
consistently described, so it must be decided which if any of the usages are correct. 

> 1 of type “hardware”: Assumptions regarding the hardware on which cFE will run 
were not documented, so that the feasibility of requirements cannot be validated. 

As can be easily seen, “missing functionality” was by far the largest category; an additional 
four defects of “extraneous” (i.e. over-specified) functionality were also found. Thus 22 out 
of 30 defect discoveries had to do with trying to adequately define a sufficiently reusable set 
of functionality to make cFE sufficiently able to support the range of future missions. This 
result makes for an interesting comparison with the earlier analysis of Swift BAT defects, in 
which such defects were only a small minority over the life of the project (namely, 4 out of 
the 24 requirements-related defects detected). This analysis highlights the difficulty of 
designing systems for maximum reusability rather than to support a specific mission. An 
implication of this result may be that we should consider creating a cFE-specific variant of 
the inspection guidelines that could specifically target such issues. 

Tailoring and comparison to other Perspective-Based inspection guidelines. To check 
the results of the tailoring process that produced the FSB perspective-based inspection 
guidelines, it is useful to compare to other tailored inspection guidelines that have been 
produced for NASA projects. Two other sets of perspectives have been created specifically 
for requirements inspections: An early version in 1994, developed by the University of 
Maryland in cooperation with GSFC, was intended to be “generic,” i.e. reusable by many 
different NASA projects. Another version was developed for the United Space Alliance 
(USA) in 2004 (also done on technology infusion funding). 

Two of the perspectives created for the Flight Software Branch (tester and developer) 
correspond to similar perspectives used in both the GSFC and USA techniques. The third 
FSB perspective (expert in hardware or other product interfaces) was unique to the Branch. 
This was not unexpected, since Branch projects by necessity have to reflect more of a 
systems engineering viewpoint so that the software produced is appropriately coordinated 
with the overall satellite mission. (In contrast, the USA techniques were designed for 
development of space station simulation software, which did not have tight constraints for 
interoperability with other hardware or software being developed in tandem.) 

At a more detailed level, we also classified the types of quality issues addressed by the 
guidelines according to the following taxonomy. These definitions are based on the quality 
standard ISO/IEC 9126 and then refined for the purposes of inspections. Text in quotes are 
taken directly from the ISO standard. 

• Clarity: The ability of the software product to support a unique implementation of the 
final software, including absence of ambiguity. 

• Feasibility: The capability of the system as specified to be implemented within the 
known capabilities and limitations of the system and its environment. Includes both 
technical aspects (is it possible to achieve the given function with the hardware 
specified) and organizational/process aspects such as prioritization or increment 
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definition (is a workable engineering strategy imaginable that could implement the 
system in reasonable cost). 

• Functionality: "The capability of the software product to provide functions which 
meet stated and implied needs when the software is used under specific conditions." 
Also includes suitability (appropriateness of the functions to fulfill user tasks), 
accuracy (correctness). 

• Maintainability: "The capability of the software product to be modified" including 
corrections, improvements, or adaptations. Includes understandability. 

• Reliability: "The capability of the software to maintain a specified level of 
performance [and required functionality] when used under specified conditions." This 
includes robustness (fault tolerance) and recoverability. 

• Testability: The capability to derive valid and useful test cases from the software 
product, supporting the ability to run these test cases in the test environment at hand. 

• Traceability: The capability to identify dependencies among parts of system artifacts. 

The FSB techniques addressed all of the above quality attributes. Most of these are in 
common with the other two sets of tailored techniques. However, the focus on 
maintainability and traceability were unique to the FSB set. This also should not be 
surprising, since the cFE project testbed on which we applied the techniques was designing a 
reusable system that could support many future missions. Thus, an emphasis on 
maintainability (making sure it was easy to be understood by developers on future missions, 
as well as appropriately modular to facilitate operating with future mission-specific 
functionality) as well as traceability (making sure that specific requirements were traced to 
the high-level goals for functionality to be provided) make good sense in the overall quality 
context of the project. 

Finally, a few of the inspection questions were added to account for particular technical areas 
that may not be as relevant for non-Flight Software projects, specifically, asynchronous event 
handling and telemetry. 

b. Which benefits from Section 8 were not observed? What negative impacts on the project, if 

any, were there? For example, “Our most experienced developer spent 20 hours of her time 

debugging the installation script before the tool could even be applied”. 

We didn’t really see any negative impacts as a result of this collaboration. The 5 hours total spent by 
Branch developers and testers on the interviews seemed a good investment considering that the 
standard was well received, at least among the group who took part in this upfront work. We are 
confident that the effort needed to respond to the feedback from the ACS group will entail a similarly 
low investment of Branch personnel effort. 


4.0 Summary and Lessons Learned 

This section is organized around the specific points we were asked to address in this report. 
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a. Evaluate the effectiveness of the technology for your application. Address any 
success criteria you had established in your proposal or at the start of the project. 

The effectiveness of the technology - as evaluated using the mainly qualitative metrics 
described in our proposal - was described in Section 3.0. Although the technique has not yet 
become a Branch standard (for reasons unrelated to the technique itself, as was also 
discussed in Section 3.0), it will be the core of whatever standard we do eventually adopt. 

We feel that we need only an incremental fixing process to make the current perspective- 
based standard suitable for adoption by the Branch. 

Both the Branch and the technology infusion team are looking for additional funding that will 
allow the more rigorous, quantitative evaluation to continue using the metrics collection 
mechanisms that were put in place as part of the infusion project. 

b. Do you plan to continue to use the technology in your group on this and future 
projects in the absence of Research Infusion funding? 

a. If so, characterize the benefits vs. resources required to use the technology 
that lead you to adopt it. 

b. If you do not plan to continue to use the technology, why not? 

As stated in previous sections, we fully intend to roll out the inspection standards in the near 
future for continued use on cFE and future Branch projects. We in the Branch have been 
quite comfortable with Dr. Shull’s observation and feedback on our pilot project and feel that 
we have gotten some value already in our interaction. 

c. Do you plan to recommend the technology more broadly; for example, inclusion in 
your division’s, or Center’s (company’s) software development practices? 

Once it has been accepted by our Change Control Board, which we envision requires only the 
minor rework already described, the standard will be used by projects across the Branch. 

d. What adoption or use challenges did you encounter? Do you foresee challenges for 
more widespread use of the technology? 

We have already addressed the primary challenge we encountered, which was due to not fully 
considering all of the technical perspectives that really should have been included. We note that the 
perspective-based approach does seem to have been effective at achieving the buy-in of the 
Command and Data Handling (C&DH) personnel who were included in the upfront work. The lesson 
to be learned from this, we feel, is that it is extremely crucial to consider all interested stakeholders in 
the tailoring process, and perhaps spending some effort on checking whether assumptions about 
which technical roles have similar/different interests in the inspection process are in fact true. 

e. What factors do you think aided the adoption and use of the technology? 
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Tailoring the inspection standard through interviews and past defect history seemed to help 
improve effectiveness and relevance and help get buy-in from Branch personnel - by 
showing that the guidelines were relevant for Branch concerns. 

f. Can you offer any suggestions for improving the technology? What changes in the 
technology would make it more usable to you? 

We have not found any major problems. Our experiences certainly show the necessity of spending 
time upfront on a thorough analysis of stakeholders and perspectives. 

We may have additional recommendations from later use of this technique. 

g. Can you offer any suggestions for improving the introduction of the technology? For 
example, help on identifying the right applications for it; changes to the training 
course; more consulting time by the vendor during technology introduction. 

No recommendations at this time. 

h. Describe the contexts, if any, for which you would recommend the technology. 
Consider not only the technical match but other factors as well, for example, the 
capabilities of the developers or the size of the project, that make the technology 
match the application. Also, list indicators for not applying the technology. 

We feel that the current version of the standard is a valuable resource for C&DH development 
activities. Due to feedback from Branch personnel, we know that it is not currently suitable for ACS 
inspections. However, we see no reason why the methodology applied to date would be unable to 
correct this if ACS personnel are included in the development of a fuller perspective-based standard. 


5.0 

Acronyms and Definitions 

ACS 

Attitude Control System 

BAT 

Burst Alert Telescope 

C&DH 

Command and Data Handling 

cFE 

Core Flight Executive 

FSB 

Flight Software Branch 

FSW 

Flight Software 

GPM 

Global Precipitation Measurement 

JWST 

James Webb Space Telescope 

RFA 

Request for Action 

SDO 

Solar Dynamics Observatory 

USA 

United Space Alliance 
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